Efficient affine merge motion vector derivation

ABSTRACT

A video processing method for efficient affine merge motion vector derivation is disclosed. In one aspect, a video processing method is provided to include partitioning a current video block into sub-blocks; deriving, for each sub-block, a motion vector, wherein the motion vector for each sub-block is associated with a position for that sub-block according to a position rule; and processing a bitstream representation of the current video block using motion vectors for the sub-blocks.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/IB2019/055592, filed on Jul. 1, 2019, which claims the priority toand benefits of International Patent Application No. PCT/CN2018/093943,filed on Jul. 1, 2018, PCT/CN2018/095568, filed on Jul. 13, 2018. Allthe aforementioned patent applications are hereby incorporated byreference in their entireties.

TECHNICAL FIELD

This patent document relates to video coding and decoding techniques,devices and systems.

BACKGROUND

In spite of the advances in video compression, digital video stillaccounts for the largest bandwidth use on the internet and other digitalcommunication networks. As the number of connected user devices capableof receiving and displaying video increases, it is expected that thebandwidth demand for digital video usage will continue to grow.

SUMMARY

This document discloses techniques that can be used in video coding anddecoding embodiments to improve performance of sub-block based coding,and in particular, when using affine motion coding mode.

In one example aspect, a video processing method is provided to includepartitioning a current block into sub-blocks; deriving, for eachsub-block, a motion vector, wherein the motion vector for each sub-blockis associated with a position for that sub-block according to a positionrule; and processing a bitstream representation of the current blockusing motion vectors for the sub-blocks.

In another aspect, a video processing method is provided to comprise:deriving, for a conversion between a current block and a bitstreamrepresentation of the current block using affine mode, motion vectors atcontrol points of the current block based on a position rule; andperforming the conversion between the current block and the bitstreamrepresentation using the motion vectors, and wherein the position rulespecifies to exclude use of non-adjacent neighboring blocks for thederiving.

In another aspect, a method of video processing is provided to comprise:determining, for a conversion between a current block and a bitstreamrepresentation of the current block, a list of affine merge candidatesfor the conversion by including merge candidates from one or moreneighboring block that satisfy a validity criterion based on positionsof the one or more neighboring blocks; and performing the conversionbetween the current block and the bitstream representation using motionvectors.

In yet another example aspect, a video encoder device that implements avideo encoding method described herein is disclosed.

In yet another representative aspect, the various techniques describedherein may be embodied as a computer program product stored on anon-transitory computer readable media. The computer program productincludes program code for carrying out the methods described herein.

In yet another representative aspect, a video decoder apparatus mayimplement a method as described herein.

The details of one or more implementations are set forth in theaccompanying attachments, the drawings, and the description below. Otherfeatures will be apparent from the description and drawings, and fromthe claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of sub-block based prediction.

FIG. 2 illustrates an example of simplified affine motion model.

FIG. 3 shows an example of affine motion vector field (MVF) persub-block.

FIG. 4 shows an example of motion vector prediction (MVP) for AF_INTERmode.

FIG. 5A and FIG. 5B depict examples of Candidates for AF_MERGE.encodingmode.

FIG. 6 shows an example process of advanced temporal motion vectorpredictor (ATMVP) motion prediction for a coding unit (CU).

FIG. 7 shows an example of one CU with four sub-blocks (A-D) and itsneighboring blocks (a-d).

FIG. 8 shows an example of optical flow trajectory in video coding.

FIGS. 9A and 9B shows an example of bi-directional optical (BIO) codingtechnique without block extension. FIG. 9A shows an example of accesspositions outside of the block and FIG. 9B shows an example of paddingused in order to avoid extra memory access and calculation.

FIG. 10 shows an example of bilateral matching.

FIG. 11 shows an example of template matching.

FIG. 12 shows an example of Unilateral motion estimation (ME) in framerate up-conversion (FRUC).

FIG. 13 illustrate an example implementation of interweaved prediction.

FIG. 14 shows an example of different positions to derive MVs fordifferent sub-blocks, where stars represent the different positions.

FIG. 15 shows examples of neighboring blocks to derive v0x and v0y.

FIG. 16A and FIG. 16B Examples of derive MVs for the affine merge modefrom left adjacent blocks coded with the affine mode (a) or from topadjacent blocks coded with the affine mode.

FIG. 17 shows an example of a neighboring block and a current blockbelonging to different coding tree unit (CTU) lines, in which an affinemerge candidate from such a neighboring block is treated as invalid.

FIG. 18 shows an example of interweaved prediction with two dividingpatterns in accordance with the disclosed technology.

FIG. 19A shows an example dividing pattern in which block is dividedinto 4×4 sub-blocks in accordance with the disclosed technology.

FIG. 19B shows an example dividing pattern in which a block is dividedinto 8×8 sub-blocks in accordance with the disclosed technology.

FIG. 19C shows an example dividing pattern in which a block is dividedinto 4×8 sub-blocks in accordance with the disclosed technology.

FIG. 19D shows an example dividing pattern in which a block is dividedinto 8×4 sub-blocks in accordance with the disclosed technology.

FIG. 19E shows an example dividing pattern in which a block is dividedinto non-uniform sub-blocks in accordance with the disclosed technology.

FIG. 19F shows another example dividing pattern in which a block isdivided into non-uniform sub-blocks in accordance with the disclosedtechnology.

FIG. 19G shows yet another example dividing pattern in which a block isdivided into non-uniform sub-blocks in accordance with the disclosedtechnology.

FIG. 20 is a block diagram of an example of a hardware platform forimplementing a visual media decoding or a visual media encodingtechnique described in the present document.

FIG. 21 is a flowchart for an example method of video processing.

FIG. 22 is a flowchart for another example method of video processing.

FIG. 23 is a flowchart for another example method of video processing.

DETAILED DESCRIPTION

Section headings are used in the present document to improve readabilityand do not limit the techniques and embodiments described in a sectionto only that section.

To improve compression ratio of video, researchers are continuallylooking for new techniques by which to encode video.

1. Introduction

This patent document is related to video/image coding technologies.Specifically, it is related to sub-block based prediction in video/imagecoding. It may be applied to the existing video coding standard likeHEVC, or the standard (Versatile Video Coding) to be finalized. It maybe also applicable to future video/image coding standards or video/imagecodec.

Brief Discussion

Sub-block based prediction is first introduced into the video codingstandard by HEVC Annex I (3D-HEVC). With sub-block based prediction, ablock, such as a Coding Unit (CU) or a Prediction Unit (PU), is dividedinto several non-overlapped sub-blocks. Different sub-block may beassigned different motion information, such as reference index or MotionVector (MV), and Motion Compensation (MC) is performed individually foreach sub-block. FIG. 1 demonstrates the concept of sub-block basedprediction.

To explore the future video coding technologies beyond HEVC, Joint VideoExploration Team (JVET) was founded by VCEG and MPEG jointly in 2015.Since then, many new methods have been adopted by JVET and put into thereference software named Joint Exploration Model (JEM).

In JEM, sub-block based prediction is adopted in several coding tools,such as affine prediction, Alternative temporal motion vector prediction(ATMVP), spatial-temporal motion vector prediction (STMVP),Bi-directional Optical flow (BIO) and Frame-Rate Up Conversion (FRUC).

2.1 Affine Prediction

In HEVC, only translation motion model is applied for motioncompensation prediction (MCP). While in the real world, there are manykinds of motion, e.g. zoom in/out, rotation, perspective motions and theother irregular motions. In the JEM, a simplified affine transformmotion compensation prediction is applied. As shown FIG. 2, the affinemotion field of the block is described by two control point motionvectors.

The motion vector field (MVF) of a block is described by the followingequation:

$\begin{matrix}\left\{ \begin{matrix}{v_{x} = {{{ax} - {by} + c} = {{\frac{\left( {v_{1x} - v_{0x}} \right)}{w}x} - {\frac{\left( {v_{1y} - v_{0y}} \right)}{w}y} + v_{0\; x}}}} \\{v_{y} = {{{bx} + {ay} + d} = {{\frac{\left( {v_{1y} - v_{0y}} \right)}{w}x} + {\frac{\left( {v_{1x} - v_{0x}} \right)}{w}y} + v_{0\; y}}}}\end{matrix} \right. & (1)\end{matrix}$

Where (v_(0x), v_(0y)) is motion vector of the top-left corner controlpoint, and (v_(1x), v_(1y)) is motion vector of the top-right cornercontrol point.

In order to further simplify the motion compensation prediction,sub-block based affine transform prediction is applied. The sub-blocksize M×N is derived as in Eq. (2), where MvPre is the motion vectorfraction accuracy ( 1/16 in JEM), (v_(2x), v_(2y)) is motion vector ofthe bottom-left control point, calculated according to Equation 1.

$\begin{matrix}\left\{ \begin{matrix}{M = {clip\ 3\ \left( {4,w,\frac{w \times {MvPre}}{\max \left( {{{abs}\left( {v_{1x} - v_{0x}} \right)},{{abs}\left( {v_{1y} - v_{0y}} \right)}} \right)}} \right)}} \\{N = {clip\ 3\ \left( {4,h,\frac{h \times {MvPre}}{\max \left( {{{abs}\; \left( {v_{2x} - v_{0x}} \right)},{{abs}\left( {v_{2y} - v_{0y}} \right)}} \right)}} \right)}}\end{matrix} \right. & (2)\end{matrix}$

After derived by Eq. (2), M and N should be adjusted downward ifnecessary to make it a divisor of w and h, respectively.

To derive motion vector of each M×N sub-block, the motion vector of thecenter sample of each sub-block, as shown in FIG. 3, is calculatedaccording to Eq. (1), and rounded to 1/16 fraction accuracy. Then themotion compensation interpolation filters are applied to generate theprediction of each sub-block with derived motion vector.

After MCP, the high accuracy motion vector of each sub-block is roundedand saved as the same accuracy as the normal motion vector.

In the JEM, there are two affine motion modes: AF_INTER mode andAF_MERGE mode. For CUs with both width and height larger than 8,AF_INTER mode can be applied. An affine flag in CU level is signalled inthe bitstream to indicate whether AF_INTER mode is used. In this mode, acandidate list with motion vector pair {(v₀, v₁)|v₀={v_(A), v_(B),v_(C)}, v₁={v_(D),v_(E)}} is constructed using the neighbour blocks. Asshown in FIG. 4, v₀ is selected from the motion vectors of the block A,B or C. The motion vector from the neighbour block is scaled accordingto the reference list and the relationship among the POC of thereference for the neighbour block, the POC of the reference for thecurrent CU and the POC of the current CU. And the approach to select v₁from the neighbour block D and E is similar. If the number of candidatelist is smaller than 2, the list is padded by the motion vector paircomposed by duplicating each of the AMVP candidates. When the candidatelist is larger than 2, the candidates are firstly sorted according tothe consistency of the neighbouring motion vectors (similarity of thetwo motion vectors in a pair candidate) and only the first twocandidates are kept. An RD cost check is used to determine which motionvector pair candidate is selected as the control point motion vectorprediction (CPMVP) of the current CU. And an index indicating theposition of the CPMVP in the candidate list is signalled in thebitstream. After the CPMVP of the current affine CU is determined,affine motion estimation is applied and the control point motion vector(CPMV) is found. Then the difference of the CPMV and the CPMVP issignalled in the bitstream.

When a CU is applied in AF_MERGE mode, it gets the first block codedwith affine mode from the valid neighbour reconstructed blocks. And theselection order for the candidate block is from left, above, aboveright, left bottom to above left as shown in FIG. 5A. If the neighbourleft bottom block A is coded in affine mode as shown in FIG. 5B, themotion vectors v₂, v₃ and v₄ of the top left corner, above right cornerand left bottom corner of the CU which contains the block A are derived.And the motion vector v₀ of the top left corner on the current CU iscalculated according to v₂, v₃ and v₄. Secondly, the motion vector v₁ ofthe above right of the current CU is calculated.

After the CPMV of the current CU v₀ and v₁ are derived, according to thesimplified affine motion model Eq (1), the MVF of the current CU isgenerated. In order to identify whether the current CU is coded withAF_MERGE mode, an affine flag is signalled in the bitstream when thereis at least one neighbour block is coded in affine mode.

2.2 ATMVP

In the alternative temporal motion vector prediction (ATMVP) method, themotion vectors temporal motion vector prediction (TMVP) is modified byfetching multiple sets of motion information (including motion vectorsand reference indices) from blocks smaller than the current CU. As shownin FIG. 6, the sub-CUs are square N×N blocks (N is set to 4 by default).

ATMVP predicts the motion vectors of the sub-CUs within a CU in twosteps. The first step is to identify the corresponding block in areference picture with a so-called temporal vector. The referencepicture is called the motion source picture. The second step is to splitthe current CU into sub-CUs and obtain the motion vectors as well as thereference indices of each sub-CU from the block corresponding to eachsub-CU, as shown in FIG. 6.

In the first step, a reference picture and the corresponding block isdetermined by the motion information of the spatial neighbouring blocksof the current CU. To avoid the repetitive scanning process ofneighbouring blocks, the first merge candidate in the merge candidatelist of the current CU is used. The first available motion vector aswell as its associated reference index are set to be the temporal vectorand the index to the motion source picture. This way, in ATMVP, thecorresponding block may be more accurately identified, compared withTMVP, wherein the corresponding block (sometimes called collocatedblock) is always in a bottom-right or center position relative to thecurrent CU.

In the second step, a corresponding block of the sub-CU is identified bythe temporal vector in the motion source picture, by adding to thecoordinate of the current CU the temporal vector. For each sub-CU, themotion information of its corresponding block (the smallest motion gridthat covers the center sample) is used to derive the motion informationfor the sub-CU. After the motion information of a corresponding N×Nblock is identified, it is converted to the motion vectors and referenceindices of the current sub-CU, in the same way as TMVP of HEVC, whereinmotion scaling and other procedures apply. For example, the decoderchecks whether the low-delay condition (i.e. the POCs of all referencepictures of the current picture are smaller than the POC of the currentpicture) is fulfilled and possibly uses motion vector MV_(x) (the motionvector corresponding to reference picture list X) to predict motionvector MV_(y) (with X being equal to 0 or 1 and Y being equal to 1-X)for each sub-CU.

3. STMVP

In this method, the motion vectors of the sub-CUs are derivedrecursively, following raster scan order. FIG. 7 illustrates thisconcept. Let us consider an 8×8 CU which contains four 4×4 sub-CUs A, B,C, and D. The neighbouring 4×4 blocks in the current frame are labelledas a, b, c, and d.

The motion derivation for sub-CU A starts by identifying its two spatialneighbours. The first neighbour is the N×N block above sub-CU A (blockc). If this block c is not available or is intra coded the other N×Nblocks above sub-CU A are checked (from left to right, starting at blockc). The second neighbour is a block to the left of the sub-CU A (blockb). If block b is not available or is intra coded other blocks to theleft of sub-CU A are checked (from top to bottom, staring at block b).The motion information obtained from the neighbouring blocks for eachlist is scaled to the first reference frame for a given list. Next,temporal motion vector predictor (TMVP) of sub-block A is derived byfollowing the same procedure of TMVP derivation as specified in HEVC.The motion information of the collocated block at location D is fetchedand scaled accordingly. Finally, after retrieving and scaling the motioninformation, all available motion vectors (up to 3) are averagedseparately for each reference list. The averaged motion vector isassigned as the motion vector of the current sub-CU.

4. BIO

Bi-directional Optical flow (BIO) is sample-wise motion refinement whichis performed on top of block-wise motion compensation for bi-prediction.The sample-level motion refinement doesn't use signalling.

Let I^((k)) be the luma value from reference k (k=0, 1) after blockmotion compensation, and ∂I^((k))/∂x, ∂I^((k))/∂y are horizontal andvertical components of the I^((k)) gradient, respectively. Assuming theoptical flow is valid, the motion vector field (v_(x), v_(y)) is givenby an equation

∂I ^((k)) /∂t+v _(x) ∂I ^((k)) /∂x+v _(y) ∂I ^((k)) /∂y=0.  (3)

Combining this optical flow equation with Hermite interpolation for themotion trajectory of each sample results in a unique third-orderpolynomial that matches both the function values I^((k)) and derivatives∂I^((k))/∂x, ∂I^((k))/∂y at the ends. The value of this polynomial att=0 is the BIO prediction:

pred_(BIO)=1/2·(I ⁽⁰⁾ +I ⁽¹⁾ +v _(x)/2·(τ₁ ∂I ⁽¹⁾ /∂x−τ ₀ ∂I ⁽⁰⁾ /∂x)+v_(y)/2·(τ₁ ∂I ⁽¹⁾ /∂y−τ ₀ ∂I ⁽⁰⁾ /∂y)).  (4)

Here, τ₀ and τ₁ denote the distances to the reference frames as shown ona FIG. 8. Distances τ₀ and τ₁ are calculated based on POC for Ref0 andRef1: τ₀=POC(current)−POC(Ref0), τ₁=POC(Ref1)−POC(current). If bothpredictions come from the same time direction (either both from the pastor both from the future) then the signs are different (i.e., τ₀·τ₁<0).In this case, BIO is applied only if the prediction is not from the sametime moment (i.e., τ₀≠τ₁), both referenced regions have non-zero motion(MVx₀,MVy₀,MVx₁,MVy₁≠0) and the block motion vectors are proportional tothe time distance (MVx₀/MVx₁=MVy₀/MVy₁=−τ₀/τ₁).

The motion vector field (v₁, v_(y)) is determined by minimizing thedifference Δ between values in points A and B (intersection of motiontrajectory and reference frame planes on FIGS. 9A and 9B). Model usesonly first linear term of a local Taylor expansion for A:

Δ=)(I ⁽⁰⁾ −I ⁽¹⁾ ₀ +v _(x)(τ₁ ∂I ⁽¹⁾ /∂x+τ ₀ ∂I ⁽⁰⁾ /∂x)+v _(y)(τ₁ ∂I⁽¹⁾ /∂y+τ ₀ ∂I ⁽⁰⁾ /∂y))  (5)

All values in the above equation depend on the sample location (i′, j′),which was omitted from the notation so far. Assuming the motion isconsistent in the local surrounding area, we minimize Δ inside the(2M+1)×(2M+1) square window Ω centered on the currently predicted point(i, j), where M is equal to 2:

$\begin{matrix}{\left( {v_{x},v_{y}} \right) = {\underset{v_{x},v_{y}}{argmin}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}\; {\Delta^{2}\left\lbrack {i^{\prime},j^{\prime}} \right\rbrack}}}} & (6)\end{matrix}$

For this optimization problem, the JEM uses a simplified approach makingfirst a minimization in the vertical direction and then in thehorizontal direction. This results in

$\begin{matrix}{\mspace{79mu} {v_{x} = {\left( {s_{1} + r} \right) > {{m?{clip}}\; 3\left( {{{- {th}}{BIO}},{- \frac{s_{3}}{\left( {s_{1} + r} \right)}}} \right)\text{:}0}}}} & (7) \\{\mspace{79mu} {v_{y} = {\left( {s_{5} + r} \right) > {{m?{clip}}\; 3\left( {{{- {th}}{BIO}},{thBIO},{- \frac{s_{6} - {v_{x}s_{2}\text{/}2}}{\left( {s_{5} + r} \right)}}} \right)\text{:}0}}}} & (8) \\{\mspace{79mu} {{{s_{1} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right)^{2}}};}\mspace{79mu} {{s_{3} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right\}}}};}{{s_{2} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}\partial_{x}}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}};}\mspace{20mu} {{s_{5} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)^{2}}};}\mspace{20mu} {s_{6} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}}}} & (9)\end{matrix}$

In order to avoid division by zero or a very small value, regularizationparameters r and m are introduced in Eq (7) and Eq (8).

r=500·4^(d-8)  (10)

m=700·4^(d-8)  (11)

Here d is bit depth of the video samples.

In order to keep the memory access for BIO the same as for regularbi-predictive motion compensation, all prediction and gradients values,I^((k)), ∂I^((k))/∂x, ∂I^((k))/∂_(y), are calculated only for positionsinside the current block. In Eq. (9), (2M+1)×(2M+1) square window SIcentered in currently predicted point on a boundary of predicted blockneeds to accesses positions outside of the block (as shown in FIG. 9A).In the JEM, values of I^((k)), ∂I^((k))/∂x, ∂I^((k))/∂y outside of theblock are set to be equal to the nearest available value inside theblock. For example, this can be implemented as padding, as shown in FIG.9B.

With BIO, it's possible that the motion field can be refined for eachsample. To reduce the computational complexity, a block-based design ofBIO is used in the JEM. The motion refinement is calculated based on 4×4block. In the block-based BIO, the values of s_(n) in Eq. (9) of allsamples in a 4×4 block are aggregated, and then the aggregated values ofs_(n) in are used to derived BIO motion vectors offset for the 4×4block. More specifically, the following formula is used for block-basedBIO derivation:

$\begin{matrix}{\mspace{79mu} {{{{{s_{1,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}\partial_{x}}} \right)^{2}}}};}\mspace{79mu} {{s_{3,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}\partial_{x}}} \right)}}}};}{{s_{2,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}}};}\mspace{20mu} {s_{5,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)^{2}}}}};}\mspace{20mu} {s_{6,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}}}}} & (12)\end{matrix}$

where b_(k) denotes the set of samples belonging to the k-th 4×4 blockof the predicted block. s_(n) in Eq (7) and Eq (8) are replaced by((s_(n,bk))>>4) to derive the associated motion vector offsets.

In some cases, MV regiment of BIO might be unreliable due to noise orirregular motion. Therefore, in BIO, the magnitude of MV regiment isclipped to a threshold value thBIO. The threshold value is determinedbased on whether the reference pictures of the current picture are allfrom one direction. If all the reference pictures of the current pictureare from one direction, the value of the threshold is set to12×2^(14-d); otherwise, it is set to 12×2^(13-d).

Gradients for BIO are calculated at the same time with motioncompensation interpolation using operations consistent with HEVC motioncompensation process (2D separable FIR). The input for this 2D separableFIR is the same reference frame sample as for motion compensationprocess and fractional position (fracX, fracY) according to thefractional part of block motion vector. In case of horizontal gradient∂I/∂x signal first interpolated vertically using BIOfilterScorresponding to the fractional position fracY with de-scaling shiftd-8, then gradient filter BIOfilterG is applied in horizontal directioncorresponding to the fractional position fracX with de-scaling shift by18-d. In case of vertical gradient ∂I/∂y first gradient filter isapplied vertically using BIOfilterG corresponding to the fractionalposition fracY with de-scaling shift d-8, then signal displacement isperformed using BIOfilterS in horizontal direction corresponding to thefractional position fracX with de-scaling shift by 18-d. The length ofinterpolation filter for gradients calculation BIOfilterG and signaldisplacement BIOfilterF is shorter (6-tap) in order to maintainreasonable complexity. Table shows the filters used for gradientscalculation for different fractional positions of block motion vector inBIO. Table shows the interpolation filters used for prediction signalgeneration in BIO.

TABLE 1 Filters for gradients calculation in BIO Fractional pel positionInterpolation filter for gradient(BIOfilterG) 0 {8, −39, −3, 46, −17, 5}1/16 {8, −32, −13, 50, −18, 5} ⅛ {7, −27, −20, 54, −19, 5} 3/16 {6, −21,−29, 57, −18, 5} ¼ {4, −17, −36, 60, −15, 4} 5/16 {3, −9, −44, 61, −15,4} ⅜ {1, −4, −48, 61, −13, 3} 7/16 {0, 1, −54, 60, −9, 2} ½ {−1, 4, −57,57, −4, 1}

TABLE 2 Interpolation filters for prediction signal generation in BIOFractional pel position Interpolation filter for predictionsignal(BIOfilterS) 0 {0, 0, 64, 0, 0, 0} 1/16 {1, −3, 64, 4, −2, 0} ⅛{1, −6, 62, 9, −3, 1} 3/16 {2, −8, 60, 14, −5, 1} ¼ {2, −9, 57, 19, −7,2} 5/16 {3, −10, 53, 24, −8, 2} ⅜ {3, −11, 50, 29, −9, 2} 7/16 {3, −11,44, 35, −10, 3} ½ {3, −10, 35, 44, −11, 3}

In the JEM, BIO is applied to all bi-predicted blocks when the twopredictions are from different reference pictures. When LIC is enabledfor a CU, BIO is disabled.

In the JEM, OBMC is applied for a block after normal MC process. Toreduce the computational complexity, BIO is not applied during the OBMCprocess. This means that BIO is only applied in the MC process for ablock when using its own MV and is not applied in the MC process whenthe MV of a neighboring block is used during the OBMC process.

2.5 FRUC

A FRUC flag is signalled for a CU when its merge flag is true. When theFRUC flag is false, a merge index is signalled and the regular mergemode is used. When the FRUC flag is true, an additional FRUC mode flagis signalled to indicate which method (bilateral matching or templatematching) is to be used to derive motion information for the block.

At encoder side, the decision on whether using FRUC merge mode for a CUis based on RD cost selection as done for normal merge candidate. Thatis the two matching modes (bilateral matching and template matching) areboth checked for a CU by using RD cost selection. The one leading to theminimal cost is further compared to other CU modes. If a FRUC matchingmode is the most efficient one, FRUC flag is set to true for the CU andthe related matching mode is used.

Motion derivation process in FRUC merge mode has two steps. A CU-levelmotion search is first performed, then followed by a Sub-CU level motionrefinement. At CU level, an initial motion vector is derived for thewhole CU based on bilateral matching or template matching. First, a listof MV candidates is generated and the candidate which leads to theminimum matching cost is selected as the starting point for further CUlevel refinement. Then a local search based on bilateral matching ortemplate matching around the starting point is performed and the MVresults in the minimum matching cost is taken as the MV for the wholeCU. Subsequently, the motion information is further refined at sub-CUlevel with the derived CU motion vectors as the starting points.

For example, the following derivation process is performed for a W×H CUmotion information derivation. At the first stage, MV for the whole W×HCU is derived. At the second stage, the CU is further split into M×Msub-CUs. The value of M is calculated as in (16), D is a predefinedsplitting depth which is set to 3 by default in the JEM. Then the MV foreach sub-CU is derived.

$\begin{matrix}{M = {\max \left\{ {4,{\min \left\{ {\frac{M}{2^{D}},\ \frac{N}{2^{D}}} \right\}}} \right\}}} & (13)\end{matrix}$

As shown in the FIG. 10, the bilateral matching is used to derive motioninformation of the current CU by finding the closest match between twoblocks along the motion trajectory of the current CU in two differentreference pictures. Under the assumption of continuous motiontrajectory, the motion vectors MV0 and MV1 pointing to the two referenceblocks shall be proportional to the temporal distances, i.e., TD0 andTD1, between the current picture and the two reference pictures. As aspecial case, when the current picture is temporally between the tworeference pictures and the temporal distance from the current picture tothe two reference pictures is the same, the bilateral matching becomesmirror based bi-directional MV.

As shown in FIG. 11, template matching is used to derive motioninformation of the current CU by finding the closest match between atemplate (top and/or left neighbouring blocks of the current CU) in thecurrent picture and a block (same size to the template) in a referencepicture. Except the aforementioned FRUC merge mode, the templatematching is also applied to AMVP mode. In the JEM, as done in HEVC, AMVPhas two candidates. With template matching method, a new candidate isderived. If the newly derived candidate by template matching isdifferent to the first existing AMVP candidate, it is inserted at thevery beginning of the AMVP candidate list and then the list size is setto two (meaning remove the second existing AMVP candidate). When appliedto AMVP mode, only CU level search is applied.

CU Level MV Candidate Set

The MV candidate set at CU level consists of:

-   -   (i) Original AMVP candidates if the current CU is in AMVP mode    -   (ii) all merge candidates,    -   (iii) several MVs in the interpolated MV field (described        later).    -   (iv) top and left neighbouring motion vectors

When using bilateral matching, each valid MV of a merge candidate isused as an input to generate a MV pair with the assumption of bilateralmatching. For example, one valid MV of a merge candidate is (MVa, refa)at reference list A. Then the reference picture refb of its pairedbilateral MV is found in the other reference list B so that refa andrefb are temporally at different sides of the current picture. If such arefb is not available in reference list B, refb is determined as areference which is different from refa and its temporal distance to thecurrent picture is the minimal one in list B. After refb is determined,MVb is derived by scaling MVa based on the temporal distance between thecurrent picture and refa, refb.

Four MVs from the interpolated MV field are also added to the CU levelcandidate list. More specifically, the interpolated MVs at the position(0, 0), (W/2, 0), (0, H/2) and (W/2, H/2) of the current CU are added.

When FRUC is applied in AMVP mode, the original AMVP candidates are alsoadded to CU level MV candidate set.

At the CU level, up to 15 MVs for AMVP CUs and up to 13 MVs for mergeCUs are added to the candidate list.

Sub-CU Level MV Candidate Set

The MV candidate set at sub-CU level consists of:

-   -   (i) an MV determined from a CU-level search,    -   (ii) top, left, top-left and top-right neighbouring MVs,    -   (iii) scaled versions of collocated MVs from reference pictures,    -   (iv) up to 4 ATMVP candidates,    -   (v) up to 4 STMVP candidates

The scaled MVs from reference pictures are derived as follows. All thereference pictures in both lists are traversed. The MVs at a collocatedposition of the sub-CU in a reference picture are scaled to thereference of the starting CU-level MV.

ATMVP and STMVP candidates are limited to the four first ones.

At the sub-CU level, up to 17 MVs are added to the candidate list.

Generation of Interpolated MV Field

Before coding a frame, interpolated motion field is generated for thewhole picture based on unilateral ME. Then the motion field may be usedlater as CU level or sub-CU level MV candidates.

First, the motion field of each reference pictures in both referencelists is traversed at 4×4 block level. For each 4×4 block, if the motionassociated to the block passing through a 4×4 block in the currentpicture (as shown in FIG. 12) and the block has not been assigned anyinterpolated motion, the motion of the reference block is scaled to thecurrent picture according to the temporal distance TD0 and TD1 (the sameway as that of MV scaling of TMVP in HEVC) and the scaled motion isassigned to the block in the current frame. If no scaled MV is assignedto a 4×4 block, the block's motion is marked as unavailable in theinterpolated motion field.

Interpolation and Matching Cost

When a motion vector points to a fractional sample position, motioncompensated interpolation is needed. To reduce complexity, bi-linearinterpolation instead of regular 8-tap HEVC interpolation is used forboth bilateral matching and template matching.

The calculation of matching cost is a bit different at different steps.When selecting the candidate from the candidate set at the CU level, thematching cost is the absolute sum difference (SAD) of bilateral matchingor template matching. After the starting MV is determined, the matchingcost C of bilateral matching at sub-CU level search is calculated asfollows:

C=SAD+w·(|MV _(x) −MV _(x) ^(s) |+|MV _(y) −MV _(y) ^(s))  (14)

where w is a weighting factor which is empirically set to 4, MV andMV^(s) indicate the current MV and the starting MV, respectively. SAD isstill used as the matching cost of template matching at sub-CU levelsearch.

In FRUC mode, MV is derived by using luma samples only. The derivedmotion will be used for both luma and chroma for MC inter prediction.After MV is decided, final MC is performed using 8-taps interpolationfilter for luma and 4-taps interpolation filter for chroma.

MV Refinement

MV refinement is a pattern based MV search with the criterion ofbilateral matching cost or template matching cost. In the JEM, twosearch patterns are supported—an unrestricted center-biased diamondsearch (UCBDS) and an adaptive cross search for MV refinement at the CUlevel and sub-CU level, respectively. For both CU and sub-CU level MVrefinement, the MV is directly searched at quarter luma sample MVaccuracy, and this is followed by one-eighth luma sample MV refinement.The search range of MV refinement for the CU and sub-CU step are setequal to 8 luma samples.

Selection of Prediction Direction in Template Matching FRUC Merge Mode

In the bilateral matching merge mode, bi-prediction is always appliedsince the motion information of a CU is derived based on the closestmatch between two blocks along the motion trajectory of the current CUin two different reference pictures. There is no such limitation for thetemplate matching merge mode. In the template matching merge mode, theencoder can choose among uni-prediction from list0, uni-prediction fromlist1 or bi-prediction for a CU. The selection is based on a templatematching cost as follows:

-   -   If costBi<=factor*min (cost0, cost1)        -   bi-prediction is used;    -   Otherwise, if cost0<=cost1        -   uni-prediction from list0 is used;    -   Otherwise,        -   uni-prediction from list1 is used;

where cost0 is the SAD of list0 template matching, cost1 is the SAD oflist1 template matching and costBi is the SAD of bi-prediction templatematching. The value of factor is equal to 1.25, which means that theselection process is biased toward bi-prediction.

The inter prediction direction selection is only applied to the CU-leveltemplate matching process.

Interweaved Prediction Examples

With interweaved prediction, a block is divided into sub-blocks withmore than one dividing patterns. A dividing pattern is defined as theway to divide a block into sub-blocks, including the size of sub-blocksand the position of sub-blocks. For each dividing pattern, acorresponding prediction block may be generated by deriving motioninformation of each sub-block based on the dividing pattern. Therefore,even for one prediction direction, multiple prediction blocks may begenerated by multiple dividing patterns. Alternatively, for eachprediction direction, only a dividing pattern may be applied.

Suppose there are X dividing patterns, and X prediction blocks of thecurrent block, denoted as P₀, P₁, . . . , P_(x-1) are generated bysub-block based prediction with the X dividing patterns. The finalprediction of the current block, denoted as P, can be generated as

$\begin{matrix}{{P\left( {x,y} \right)} = \frac{\Sigma_{i = 0}^{X - 1}{w_{i}\left( {x,y} \right)} \times {P_{i}\left( {x,y} \right)}}{\Sigma_{i = 0}^{X - 1}{w_{i}\left( {x,y} \right)}}} & (15)\end{matrix}$

where (x, y) is the coordinate of a pixel in the block and w_(i)(x, y)is the weighting value of P_(i). Without losing generalization, it issupposed that Σ_(i=0) ^(X-1)w_(i)(x,y)=(1<<N) wherein N is anon-negative value. FIG. 13 shows an example of interweaved predictionwith two dividing patterns.

3. Example Problems Solved by the Described Embodiments

There are two potential drawbacks of the affine merge MV derivationprocess as shown in FIG. 5.

First, the coordinate of the left-top point of a CU and the size of theCU must be stored by each 4×4 block belonging to the CU. Thisinformation is not required to be stored in HEVC

Second, the decoder must access MVs of 4×4 blocks not adjacent to thecurrent CU. In HEVC, the decoder only needs to access MVs of 4×4 blocksadjacent to the current CU.

4. Examples of Embodiments

We propose several methods to further improve sub-block basedprediction, including the interweaved prediction and the affine merge MVderivation process.

The listing of techniques and embodiments below should be considered asexamples to explain general concepts. Furthermore, these techniques canbe combined to operate together during video encoding, orcorrespondingly during decoding, process. Note that here the term“encoding” includes “transcoding” in which source video in anon-compressed format is encoded into another coded format.

MV Derivation of Sub-Blocks

-   -   1. In one embodiment, the MV for a sub-block is derived for the        center of the sub-block.        -   a. Alternatively, the MV for a sub-block is derived for any            position inside the sub-block, which may not be at the            center of the sub-block.        -   b. Alternatively, the position for which MV is derived may            be different for each sub-block. (The position is relative            to each sub-block)        -   c. the position for which MV is derived may depend on the            location of the sub-block. FIG. 14 shows an example.        -   d. Denote the sub-block size by M×N, wherein the center            position could be defined as ((M>>1)+a)×((N>>1)+b) wherein            a, b could be 0 or −1.

FIG. 14 shows an example of different positions to derive MVs fordifferent sub-blocks. Stars represent the positions. As can be seen,various different positions may be used for MV derivation.

Efficient Affine Merge MV Derivation

-   -   2. In one embodiment, the MVs at control points (such as mv0 at        the top-left point and mv1 at the top-right point) are derived        only with the information of the adjacent neighbouring blocks,        in the affine merge MV derivation process. In one example, the        coordinate of the left-top point and the size of a neighbouring        CU, and MVs of 4×4 blocks not adjacent to the current CU are not        needed to derive the MVs of the current CU with the affine merge        mode.        -   a. In one embodiment, the affine parameters (such as a, b, c            and d for the four-parameter affine mode in eq (1)) are            stored in each block coded with the affine mode (including            affine inter-mode and affine merge mode).            -   i. If a block is coded with the affine merge mode, it                inherits the four parameters from a neighbouring block                coded with the affine mode.            -   ii. In one example, the four parameters are different                for list 0 and list1.            -   iii. In one examples, parameters for both reference                picture lists may be stored. Alternatively, only a set                of affine parameters may be stored even for                bi-prediction. Alternatively, for multiple hypothesis, 2                sets of affine parameters may be stored and each one                corresponds to one reference picture list for                bi-prediction.        -   b. In one embodiment, only partial of a set of affine            parameters (e.g., the two parameters (a and b) in eq (1) for            the four-parameter affine) are stored in each block coded            with the affine mode (including affine inter-mode and affine            merge mode). If a block is coded with the affine merge mode,            it inherits the stored partial parameters from a            neighbouring block coded with the affine mode.            -   i. In one example, different reference pictures or                different reference picture lists may store all the                related partial affine parameters.            -   ii. The two parameters are different for list 0 and                list1.            -   iii.        -   c. In one embodiment, v_(0x) and v_(0y) (also denoted as c            and d) in eq (1) are derived from blocks adjacent to the to            the top-left corner of the current block. In the following            examples, it is supposed the current block is merged to            neighbouring block G coded with affine mode.            -   i. In one example, three neighbouring blocks R, S and T                as shown in FIG. 15 are used to derive (v_(0x), v_(0y)).                MVs in the three blocks are noted as MV(R), MV(S) and                MV(T).                -   (a) In one example, (v_(0x), v_(0y)) is set equal to                    MV(X) (X can be R, S or T), if X is inter-coded.                -   (b) In one example, (v_(0x), v_(0y)) is set equal to                    the average of MV(R), MV(S) and MV(T), if R, S and T                    are inter-coded.                -   (c) In one example, (v_(0x), v_(0y)) is set equal to                    the average of MV(X) and MV(Y) (X and Y can be R, S                    or T), if X and Y are inter-coded.                -   (d) In one example, (voy, v_(0y)) is set equal to                    MV(X) and MV(X) should refer to the same reference                    of block G.            -   ii. In one example, (voy, v_(0y)) is derived from the MV                of temporal neighbouring blocks.            -   iii. In one example, (voy, v_(0y)) is scaled to the                reference of block G.        -   d. In one embodiment, the MVs of a block coded with the            affine merge mode are derived from S (S=2 for four-parameter            affine mode, 3 for six-parameter affine mode) left adjacent            blocks coded with the affine mode. FIG. 18A shows an            example. L0 and L1 are two left adjacent blocks coded with            the affine mode. Δ is the distance between the two left            adjacent blocks. The motion vectors of the two blocks are            (mvL₀ ^(x), mvL₀ ^(y)) and (mvL₁ ^(x), mvL₁ ^(y)),            respectively. (mv₀ ^(x), mv₀ ^(y)) is the MV at the top-left            control point of the current block (a. k. a. (v_(0y),            v_(0y)) in eq(1). The y-distance between one of the two            blocks (for example L0) and the top-left control point is            noted as Φ. It should be noted that the distance can be            measured from the top, the middle or the bottom of the            block. In FIG. 16A, it is measured from the bottom.            -   i. In one example, a and b in eq. (1) can be derived as

a=(mvL ₁ ^(y) −mvL ₀ ^(y))/Δ,b=−(mvL ₁ ^(x) −mvL ₀ ^(x))/Δ.

-   -   ii. Δ can be a fixed number.        -   (a) It can be in a form of 2^(N), such as 1, 4, 8, 16 etc.            In this case, the division operation to calculate a and b            above can be implemented as a shift operation.            -   iii. Δ can be a number depending on the height of the                block.            -   iv. Δ can be derived as the maximum length satisfying                that all left adjacent blocks between L0 and L1 (both                included) are coded with the affine mode and share the                same reference picture.            -   v. (mv₀ ^(x), mv₀ ^(y)) can be derived as

mv ₀ ^(x) =mvL ₀ ^(x) +bΦ,mv ₀ ^(y) =mvL ₀ ^(y) −aΦ.

-   -   vi. If Φ is the y-distance between L1 and the top-left control        point, then (mv₀ ^(x), mv₀ ^(y)) can be derived as

mv ₀ ^(x) =mvL ₁ ^(x) +bΦ,mv ₀ ^(y) =mvL ₁ ^(y) −aΦ.

-   -   e. In one embodiment, the MVs of a block coded with the affine        merge mode are derived from S (S=2 for four-parameter affine        mode, 3 for six-parameter affine mode) top adjacent blocks coded        with the affine mode. FIG. 16B shows an example. T0 and T1 are        two top adjacent blocks coded with the affine mode. Δ is the        distance between the two top adjacent blocks. The motion vectors        of the two blocks are (mvT₀ ^(x), mvT₀ ^(y)) and (mvT₁ ^(x),        mvT₁ ^(y)), respectively. (mv₀ ^(x), mv₀ ^(y)) is the MV at the        top-left control point of the current block (a. k. a. (v_(0x),        v_(0y)) in eq(1). The x-distance between one of the two blocks        (for example T0) and the top-left control point is noted as Φ.        It should be noted that the distance can be measured from the        left, the middle or the right of the block. In FIG. 16B, it is        measured from the right.        -   i. In one example, a and b in eq. (1) can be derived as

a=(mvT ₁ ^(x) −mvT ₀ ^(x))/Δ,b=(mvT ₁ ^(y) −mvT ₀ ^(y))/Δ.

-   -   ii. Δ can be a fixed number.        -   (a) It can be in a form of 2^(N), such as 1, 4, 8, 16 etc.            In this case, the division operation to calculate a and b            above can be implemented as a shift operation.            -   iii. Δ can be a number depending on the height of the                block.            -   iv. Δ can be derived as the maximum length satisfying                that all top adjacent blocks between T0 and T1 (both                included) are coded with the affine mode and share the                same reference picture.            -   v. (mv₀ ^(x), mv₀ ^(y)) can be derived as

mv ₀ ^(x) =mvT ₀ ^(x) −aΦ,mv ₀ ^(y) =mvT ₀ ^(y) −bΦ.

-   -   vi. If Φ is the x-distance between T1 and the top-left control        point, then (mv₀ ^(x), mv₀ ^(y)) can be derived as

mv ₀ ^(x) =mvT ₁ ^(x) −aΦ,mv ₀ ^(y) =mvT ₁ ^(y) −bΦ.

FIGS. 16A and 16B show examples of derive MVs for the affine merge modefrom left adjacent blocks coded with the affine mode (FIG. 16A) or fromtop adjacent blocks coded with the affine mode (FIG. 16B).

-   -   f. MVs of a block coded with affine merge mode may be derived        from non-adjacent blocks coded with affine mode.    -   g. Which adjacent blocks are used to derive MVs of a block coded        with the affine merge mode may depend on the block shape.        -   i. For a block with size M×N and M>N, the MVs of a block            coded with the affine merge mode are derived from top            adjacent blocks coded with the affine mode.        -   ii. For a block with size M×N and M<N, the MVs of a block            coded with the affine merge mode are derived from left            adjacent blocks coded with the affine mode.        -   iii. For a block with size M×N and M=N, the MVs of a block            coded with the affine merge mode are derived from blocks            adjacent to the to the top-left corner of the current block.    -   3. In one embodiment, whether an affine merge candidate from a        neighboring block is a valid affine merge candidate depends on        the location of the neighboring block.    -   a. In one example, an affine merge candidate from a neighboring        block is treated as invalid (not be put into the merge candidate        list) if the neighboring block belongs to a Coding Tree Unit        (CTU) (e.g., a Largest CU (LCU)) different from the current CTU.    -   b. Alternatively, an affine merge candidate from a neighboring        block is treated as invalid (not be put into the merge candidate        list) if the neighboring block belongs to a CTU line different        from the current CTU line, as shown in FIG. 17.    -   c. Alternatively, an affine merge candidate from a neighbouring        block is treated as invalid (not be put into the merge candidate        list) if the neighbouring block belongs to a slice different        from the slice.    -   d. Alternatively, an affine merge candidate from a neighbouring        block is treated as invalid (not be put into the merge candidate        list) if the neighbouring block belongs to a tile different from        the tile.

FIG. 17 shows an example of a neighboring block and a current blockbelonging to different CTU lines. In this example, an affine mergecandidate from a neighboring block is treated as invalid (not be putinto the merge candidate list) if the neighboring block belongs to a CTUline different from the current CTU line.

Examples of Interweaved Prediction

FIG. 18 shows an example of interweaved prediction with two dividingpatterns in accordance with the disclosed technology. A current block1300 can be divided into multiple patterns. For example, as shown inFIG. 18, the current block is divided into both Pattern 0 (1301) andPattern 1 (1302). Two prediction blocks, P₀ (1303) and P₁ (1304), aregenerated. A final prediction block P (1305) of the current block 1300can be generated by computing a weighted sum of P₀ (1303) and P₁ (1304).

More generally, given X dividing patterns, X prediction blocks of thecurrent block, denoted as P₀, P₁, . . . , P_(X-1), can be generated bysub-block based prediction with the X dividing patterns. The finalprediction of the current block, denoted as P, can be generated as

$\begin{matrix}{{P\left( {x,y} \right)} = \frac{\Sigma_{i = 0}^{X - 1}{w_{i}\left( {x,y} \right)} \times {P_{i}\left( {x,y} \right)}}{\Sigma_{i = 0}^{X - 1}{w_{i}\left( {x,y} \right)}}} & {{Eq}.\mspace{11mu} (15)}\end{matrix}$

Here, (x, y) is the coordinate of a pixel in the block and w_(i)(x, y)is the weighting value of P_(i). By the way of example, and not bylimitation, the weights can be expressed as:

Σ_(i=0) ^(X-1) w _(i)(x,y)=(1<<N)  Eq. (16)

N is a non-negative value. Alternatively, the bit-shifting operation inEq. (16) can also be expressed as:

Σ_(i=0) ^(X-1) w _(i)(x,y)=2^(N)  Eq. (17)

The sum of the weights being a power of two allows a more efficientcomputation of the weighted sum P by performing a bit-shifting operationinstead of a floating-point division.

Dividing patterns can have different shapes, or sizes, or positions ofsub-blocks. In some embodiments, a dividing pattern may includeirregular sub-block sizes. FIGS. 19A-G show several examples of dividingpatterns for a 16×16 block. In FIG. 19A, a block is divided into 4×4sub-blocks in accordance with the disclosed technology. This pattern isalso used in JEM. FIG. 19B shows an example of a block being dividedinto 8×8 sub-blocks in accordance with the disclosed technology. FIG.19C shows an example of the block being divided into 8×4 sub-blocks inaccordance with the disclosed technology. FIG. 19D shows an example ofthe block being divided into 4×8 sub-blocks in accordance with thedisclosed technology. In FIG. 19E, a portion of the block is dividedinto 4×4 sub-blocks in accordance with the disclosed technology. Thepixels at block boundaries are divided in smaller sub-blocks with sizeslike 2×4, 4×2 or 2×2. Some sub-blocks may be merged to form largersub-blocks. FIG. 19F shows an example of adjacent sub-blocks, such as4×4 sub-blocks and 2×4 sub-blocks, that are merged to form largersub-blocks with sizes like 6×4, 4×6 or 6×6. In FIG. 19G, a portion ofthe block is divided into 8×8 sub-blocks. The pixels at block boundariesare divided in smaller sub-blocks with sizes like 8×4, 4×8 or 4×4instead.

The shapes and sizes of sub-blocks in sub-block based prediction can bedetermined based on the shape and/or size of the coding block and/orcoded block information. For example, in some embodiments, thesub-blocks have a size of 4×N (or 8×N, etc.) when the current block hasa size of M×N. That is, the sub-blocks have the same height as thecurrent block. In some embodiments, the sub-blocks have a size of M×4(or M×8, etc.) when the current block has a size of M×N. That is, thesub-blocks have the same width as the current block. In someembodiments, the sub-blocks have a size of A×B with A>B (e.g., 8×4) whenthe current block has a size of M×N, where M>N. Alternatively, thesub-blocks can have the size of B×A (e.g. 4×8).

In some embodiments, the current block has a size of M×N. The sub-blockshave a size of A×B when M×N<=T (or Min(M, N)<=T, or Max(M, N)<=T, etc.),and the sub-blocks have a size of C×D when M×N>T (or Min(M, N)>T, orMax(M, N)>T, etc.), where A<=C and B<=D. For example, if M×N<=256,sub-blocks can be in a size of 4×4. In some implementations, thesub-blocks have a size of 8×8.

In some embodiments, whether to apply interweaved prediction can bedetermined based on the inter-prediction direction. For example, in someembodiments, the interweaved prediction may be applied for bi-predictionbut not for uni-prediction. As another example, when multiple-hypothesisis applied, the interweaved prediction may be applied for one predictiondirection when there are more than one reference blocks.

In some embodiments, how to apply interweaved prediction may also bedetermined based on the inter-prediction direction. In some embodiments,a bi-predicted block with sub-block based prediction is divided intosub-blocks with two different dividing patterns for two differentreference lists. For example, a bi-predicted block is divided into 4×8sub-blocks as shown in FIG. 19D when predicted from reference list 0(L0). The same block is divided into 8×4 sub-blocks as shown in FIG. 19Cwhen predicted from reference list 1 (L1). The final prediction P iscalculated as

$\begin{matrix}{{P\left( {x,y} \right)} = \frac{{{w^{0}\left( {x,y} \right)} \times {P^{0}\left( {x,y} \right)}} + {{w^{1}\left( {x,y} \right)} \times {P^{1}\left( {x,y} \right)}}}{{w^{0}\left( {x,y} \right)} + {w^{1}\left( {x,y} \right)}}} & {{Eq}.\mspace{11mu} (18)}\end{matrix}$

Here, P⁰ and P¹ are predictions from L0 and L1, respectively. w⁰ and w¹are weighting values for L0 and L1, respectively. As shown in Eq. (16),the weighting values can be determined as: w⁰ (x, y)+w¹(x, y)=1<<N(wherein N is non-negative integer value). Because fewer sub-blocks areused for prediction in each direction (e.g., 4×8 sub-blocks as opposedto 8×8 sub-blocks), the computation requires less bandwidth as comparedto the existing sub-block based methods. By using larger sub-blocks, theprediction results are also less susceptible to noise interference.

In some embodiments, a uni-predicted block with sub-block basedprediction is divided into sub-blocks with two or more differentdividing patterns for the same reference list. For example, theprediction for list L (L=0 or 1) P^(L) is calculated as

$\begin{matrix}{{P^{L}\left( {x,y} \right)} = \frac{\Sigma_{i = 0}^{{XL} - 1}{w_{i}^{L}\left( {x,y} \right)} \times {P_{i}^{L}\left( {x,y} \right)}}{\Sigma_{i = 0}^{{XL} - 1}{w_{i}^{L}\left( {x,y} \right)}}} & {{Eq}.\mspace{11mu} (19)}\end{matrix}$

Here XL is the number of dividing patterns for list L. P_(i) ^(L)(x, y)is the prediction generated with the i^(th) dividing pattern and w_(i)^(L)(x, y) is the weighting value of P_(i) ^(L)(x, y). For example, whenXL is 2, two dividing patterns are applied for list L. In the firstdividing pattern, the block is divided into 4×8 sub-blocks as shown inFIG. 19D. In the second dividing pattern, the block is divided into 8×4sub-blocks as shown in FIG. 19D.

In some embodiments, a bi-predicted block with sub-block basedprediction is considered as a combination of two uni-predicted blockfrom L0 and L1 respectively. The prediction from each list can bederived as described in the above example. The final prediction P can becalculated as

$\begin{matrix}{{P\left( {x,y} \right)} = \frac{{a*\frac{\begin{matrix}{\Sigma_{i = 0}^{{XL0} - 1}{w_{i}^{0}\left( {x,y} \right)} \times} \\{P_{i}^{0}\left( {x,y} \right)}\end{matrix}}{\Sigma_{i = 0}^{{XL0} - 1}{w_{i}^{0}\left( {x,y} \right)}}} + {b*\frac{\begin{matrix}{\Sigma_{i = 0}^{{XL1} - 1}{w_{i}^{1}\left( {x,y} \right)} \times} \\{P_{i}^{1}\left( {x,y} \right)}\end{matrix}}{\Sigma_{i = 0}^{{XL1} - 1}{w_{i}^{1}\left( {x,y} \right)}}}}{a + b}} & {{Eq}.\mspace{11mu} (20)}\end{matrix}$

Here parameters a and b are two additional weights applied to the twointernal prediction blocks. In this specific example, both a and b canbe set to 1. Similar to the example above, because fewer sub-blocks areused for prediction in each direction (e.g., 4×8 sub-blocks as opposedto 8×8 sub-blocks), the bandwidth usage is better than or on par withthe existing sub-block based methods. At the same time, the predictionresults can be improved by using larger sub-blocks.

In some embodiments, a single non-uniform pattern can be used in eachuni-predicted block. For example, for each list L (e.g., L0 or L1), theblock is divided into a different pattern (e.g., as shown in FIG. 19E orFIG. 19F). The use of a smaller number of sub-blocks reduces the demandon bandwidth. The non-uniformity of the sub-blocks also increasesrobustness of the prediction results.

In some embodiments, for a multiple-hypothesis coded block, there can bemore than one prediction blocks generated by different dividing patternsfor each prediction direction (or reference picture list). Multipleprediction blocks can be used to generate the final prediction withadditional weights applied. For example, the additional weights may beset to 1/M wherein M is the total number of generated prediction blocks.

In some embodiments, the encoder can determine whether and how to applythe interweaved prediction. The encoder then can transmit informationcorresponding to the determination to the decoder at a sequence level, apicture level, a view level, a slice level, a Coding Tree Unit (CTU)(also known as a Largest Coding Unit (LCU)) level, a CU level, a PUlevel, a Tree Unit (TU) level, or a region level (which may includemultiple CUs/PUs/Tus/LCUs). The information can be signaled in aSequence Parameter Set (SPS), a view parameter set (VPS), a PictureParameter Set (PPS), a Slice Header (SH), a CTU/LCU, a CU, a PU, a TU,or a first block of a region.

In some implementations, the interweaved prediction applies to existingsub-block methods like the affine prediction, ATMVP, STMVP, FRUC, orBIO. In such cases, no additional signaling cost is needed. In someimplementations, new sub-block merge candidates generated by theinterweaved prediction can be inserted into a merge list, e.g.,interweaved prediction+ATMVP, interweaved prediction+STMVP, interweavedprediction+FRUC etc.

In some embodiments, the dividing patterns to be used by the currentblock can be derived based on information from spatial and/or temporalneighboring blocks. For example, instead of relying on the encoder tosignal the relevant information, both encoder and decoder can adopt aset of predetermined rules to obtain dividing patterns based on temporaladjacency (e.g., previously used dividing patterns of the same block) orspatial adjacency (e.g., dividing patterns used by neighboring blocks).

In some embodiments, the weighting values w can be fixed. For example,all dividing patterns can be weighted equally: w_(i)(x, y)=1. In someembodiments, the weighting values can be determined based on positionsof blocks as well as the dividing patterns used. For example, w_(i)(x,y) may be different for different (x, y). In some embodiments, theweighting values may further depend on the sub-block prediction basedcoding techniques (e.g., affine, or ATMVP) and/or other codedinformation (e.g., skip or non-skip modes, and/or MV information).

In some embodiments, the encoder can determine the weighting values, andtransmit the values to the decoder at sequence level, picture level,slice level, CTU/LCU level, CU level, PU level, or region level (whichmay include multiple CUs/PUs/Tus/LCUs). The weighting values can besignaled in a Sequence Parameter Set (SPS), a Picture Parameter Set(PPS), a Slice Header (SH), a CTU/LCU, a CU, a PU, or a first block of aregion. In some embodiments, the weighting values can be derived fromthe weighting values of a spatial and/or temporal neighboring block.

It is noted that the interweaved prediction techniques disclosed hereincan be applied in one, some, or all coding techniques of sub-block basedprediction. For example, the interweaved prediction techniques can beapplied to affine prediction, while other coding techniques of sub-blockbased prediction (e.g., ATMVP, STMVP, FRUC or BIO) do not use theinterweaved prediction. As another example, all of affine, ATMVP, andSTMVP apply the interweaved prediction techniques disclosed herein.

FIG. 20 is a block diagram of an example video bitstream processingapparatus 2000. The apparatus 2000 may be used to implement one or moreof the methods described herein. The apparatus 2000 may be embodied in asmartphone, tablet, computer, Internet of Things (IoT) receiver, and soon. The apparatus 2000 may include one or more processors 2002, one ormore memories 2004 and video processing hardware 2006. The processor(s)2002 may be configured to implement one or more methods described in thepresent document. The memory (memories) 2004 may be used for storingdata and code used for implementing the methods and techniques describedherein. The video processing hardware 2006 may be used to implement, inhardware circuitry, some techniques described in the present document.Note that partial or full externality of memory 2004 and circuitry 2006from the processor 2002 electronics is optional and is an implementationchoice.

FIG. 21 shows a flowchart for an example method 2100 for videoprocessing. The method 2100 includes partitioning (2102) a current blockinto sub-blocks. The method 2100 further includes deriving (2104), foreach sub-block, a motion vector, wherein the motion vector for eachsub-block is associated with a position for that sub-block according toa position rule. The method 2100 further includes processing (2106) abitstream representation of the current block using motion vectors forthe sub-blocks.

FIG. 22 is a flowchart for an example method 2200 for video processing.The method 2200 includes deriving (2202), for a conversion between acurrent block and a bitstream representation of the current block usingaffine mode, motion vectors at control points of the current block basedon a position rule. The method 2200 further includes performing (2204)the conversion between the current block and the bitstreamrepresentation using the motion vectors. In some implementations, theposition rule may specify to exclude the use of non-adjacent neighboringblocks for the deriving. In some implementations, the motion vectors maybe derived without using information of a neighboring coding unit thatincludes at least one non-adjacent 4×4 block of the current block. Insome implementations, the method further includes storing and reusing atleast some affine parameters of a previously converted neighboringblock. In some implementations, the storing and the reusing of at leastsome affine parameters can be performed in two steps separately fromeach other.

FIG. 23 is a flowchart for an example method 2300 for video processing.The method 2300 includes determining (2302), for a conversion between acurrent block and a bitstream representation of the current block, alist of affine merge candidates for the conversion by including mergecandidates from one or more neighboring block that satisfy a validitycriterion based on positions of the one or more neighboring blocks. Themethod 2300 further includes performing (2304) the conversion betweenthe current block and the bitstream representation using the motionvectors.

Additional features and embodiments of the above-describedmethods/techniques are described below using a clause-based descriptionformat.

1. A method of video processing (e.g., method 2100 shown in FIG. 21),comprising: partitioning a current block into sub-blocks; deriving, foreach sub-block, a motion vector, wherein the motion vector for eachsub-block is associated with a position for that sub-block according toa position rule; and processing a bitstream representation of thecurrent block using motion vectors for the sub-blocks.

2. The method of clause 1, wherein the position rule specifies that theposition is a center of a corresponding sub-block.

3. The method of clause 2, wherein the corresponding sub-block has asize M×N and the center is defined as ((M>>1)+a)×((N>>1)+b), wherein Mand N are natural numbers and a, b is 0 or −1.

4. The method of clause 1, wherein the position rule specifies that theposition is a non-center position of a corresponding sub-block.

5. The method of clause 1, wherein positions specified by the positionrule result in motion vectors being derived at different positions indifferent sub-blocks.

6. A video processing method (e.g., method 2200 shown in FIG. 22),comprising: deriving, for a conversion between a current block and abitstream representation of the current block using affine mode, motionvectors at control points of the current block based on a position rule;and performing the conversion between the current block and thebitstream representation using the motion vectors, and wherein theposition rule specifies to exclude use of non-adjacent neighboringblocks for the deriving.

7. The method of clause 6, wherein motion vectors are derived withoutusing information of a neighboring coding unit that includes at leastone non-adjacent 4×4 block of the current block.

8. The method of clause 7, further including: storing and reusing atleast some affine parameters of a previously converted neighboringblock.

9. The method of clause 8, wherein the current block inherits the atleast some affine parameters from at a neighboring block coded in affinemode.

10. The method of clause 8, wherein the at least some affine parametersare different for list 0 and list 1 reference frames.

11. The method of clause 8, wherein the at least some affine parameterscomprise two sets, each for one of a multiple hypothesis referencepicture list.

12. The method of any of clauses 8 to 11, wherein the at least someaffine parameters comprise two of four affine parameters.

13. The method of clause 6, wherein a motion vector (v_(0x), v_(0x)) ofa top-left corner of the current block is derived from blocks adjacentto the top-left corner of the current block and the current block ismerged to a neighboring block coded with the affine mode.

14. The method of clause 13, further including: using three neighboringblocks R, S, and T having corresponding motion vectors MV(R), MV(S), andMV(T), respectively, to derive the motion vector (v_(0x), v_(0x)), andwherein the motion vector (v_(0x), v_(0x)) is set equal to MV(X) and Xis R, S or T, if X is inter-coded.

15. The method of clause 13, further including: using three neighboringblocks R, S, and T having corresponding motion vectors MV(R), MV(S), andMV(T), respectively, to derive the motion vector (v_(0x), v_(0x)), andwherein the motion vector (v_(0x), v_(0x)) is set equal to an average ofMV(R), MV(S) and MV(T), if R, S and T are inter-coded.

16. The method of clause 13, further including: using three neighboringblocks R, S, and T having corresponding motion vectors MV(R), MV(S), andMV(T), respectively, to derive the motion vector (v_(0x), v_(0x)), andwherein the motion vector (v_(0x), v_(0x)) is set equal to an average ofMV(X) and MV(Y) and X and Y is R, S, or T, if X and Y are inter-coded.

17. The method of clause 13, wherein the motion vector (v_(0x), v_(0x))is derived from motion vectors of temporal neighboring blocks.

18. The method of clause 13, wherein the motion vector (v_(0x), v_(0x))is scaled to a reference of the neighboring block.

19. The method of clause 6, wherein the motion vectors are derived fromleft adjacent blocks coded with the affine mode.

20. The method of clause 6, wherein motion vectors of a block coded withthe affine mode are derived from S top adjacent blocks coded with theaffine mode, S being equal to 2 for four parameter affine mode.

21. The method of clause 20, wherein a distance between two top adjacentblocks is a fixed number in a form of 2^(N), N being an integer.

22. The method of clause 20, wherein a distance between the top adjacentblocks depends on a height of the block coded with the affine mode.

23. The method of clause 20, wherein a distance between the top adjacentblocks is derived as a maximum length satisfying that all top adjacentblocks are coded with the affine mode and share a same referencepicture.

24. The method of clause 20, wherein a motion vector (mv₀ ^(x), mv₀^(y)) at a top-left control point of the current block is derived as i)mv₀ ^(x)=mvT₀ ^(x)) and mv₀ ^(y)=mvT₀ ^(y)−bΦ or ii) mv₀ ^(x)=mvT₁^(x)−aΦ, mv₀ ^(y)=mvT₁ ^(y)−bΦ, Φ being a distance between the top-leftcontrol point and one of two top adjacent blocks T0 and T1 coded withthe affine mode.

25. The method of clause 6, wherein the current block has a size of M×Npixels, where M and N are integers, and the motion vectors are derivedfrom left side adjacent blocks when M<N.

26. The method of clause 6, wherein the current block has a size of M×Npixels, where M and N are integers, and the motion vectors are derivedfrom top side adjacent blocks when M>N.

27. The method of clause 6, wherein the current block has a size of M×Npixels, where M and N are integers, and the motion vectors are derivedfrom blocks adjacent to top-left corner when M=N.

28. A method of video processing (e.g., method 2300 shown in FIG. 23),comprising: determining, for a conversion between a current block and abitstream representation of the current block, a list of affine mergecandidates for the conversion by including merge candidates from one ormore neighboring block that satisfy a validity criterion based onpositions of the one or more neighboring blocks; and performing theconversion between the current block and the bitstream representationusing motion vectors.

29. The method of clause 28, wherein a neighboring block is from a CTU(coding tree unit) that is different from a current CTU, and wherein anaffine merge mode candidate from the neighboring block is invalid.

30. The method of clause 28, wherein a current CTU belongs to a currentCTU line, wherein the neighboring block belongs to a CTU line differentfrom the current CTU line, and wherein the affine merge mode candidatefrom the neighboring block is invalid.

31. The method of clause 28, wherein the current block belongs to acurrent slice, wherein the neighboring block belongs to a slicedifferent from the current slice, and wherein the affine merge modecandidate from the neighboring block is invalid.

32. The method of clause 28, wherein the current block belongs to acurrent tile, wherein the neighboring block belongs to a tile differentfrom the current tile, and wherein the affine merge mode candidate fromthe neighboring block is invalid.

33. The method of any of above clauses wherein the conversion includedgenerating the bitstream representation from the current block.

34. The method of any of above clauses wherein the conversion includedgenerating the current block from the bitstream representation.

35. A video decoding apparatus comprising a processor configured toimplement a method recited in one or more of clauses 1 to 34.

36. A video encoding apparatus comprising a processor configured toimplement a method recited in one or more of clauses 1 to 34.

37. A computer-readable program medium having code stored thereupon, thecode comprising instructions that, when executed by a processor, causingthe processor to implement a method recited in one or more of clauses 1to 34.

From the foregoing, it will be appreciated that specific embodiments ofthe presently disclosed technology have been described herein forpurposes of illustration, but that various modifications may be madewithout deviating from the scope of the invention. Accordingly, thepresently disclosed technology is not limited except as by the appendedclaims.

The disclosed and other embodiments, modules and the functionaloperations described in this document can be implemented in digitalelectronic circuitry, or in computer software, firmware, or hardware,including the structures disclosed in this document and their structuralequivalents, or in combinations of one or more of them. The disclosedand other embodiments can be implemented as one or more computer programproducts, i.e., one or more modules of computer program instructionsencoded on a computer readable medium for execution by, or to controlthe operation of, data processing apparatus. The computer readablemedium can be a machine-readable storage device, a machine-readablestorage substrate, a memory device, a composition of matter effecting amachine-readable propagated signal, or a combination of one or morethem. The term “data processing apparatus” encompasses all apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal, thatis generated to encode information for transmission to suitable receiverapparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this document can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random-access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of non-volatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto optical disks; and CD ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in, special purposelogic circuitry.

While this patent document contains many specifics, these should not beconstrued as limitations on the scope of any invention or of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of particular inventions. Certain features thatare described in this patent document in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

What is claimed is:
 1. A method of coding video data, comprising:determining, for a conversion between a current video block of a videoand a bitstream representation of the video, motion vectors at controlpoints (CPMV) of the current video block based on a rule, wherein therule specifies to exclude using of non-adjacent neighboring block fromone or more neighboring blocks of the current video block; andperforming the conversion between the current video block and thebitstream representation based on the motion vectors.
 2. The method ofclaim 1, wherein the rule further specifies to exclude using of invalidneighboring block from the one or more neighboring blocks based onpositions of the one or more neighboring blocks.
 3. The method of claim2, wherein the current video block belongs to a current slice, and aneighboring block of the one or more neighboring blocks is invalid in acase that the neighboring block belongs to a slice different from thecurrent slice.
 4. The method of claim 2, wherein the current video blockbelongs to a current tile, and a neighboring block of the one or moreneighboring blocks is invalid in a case that the neighboring blockbelongs to a tile different from the current tile.
 5. The method ofclaim 1, wherein the current video block includes multiple sub-blocks,and performing the conversion comprising: determining a motion vectorfor each sub-block of the multiple sub-blocks based on the CPMV and aspecific position of the corresponding sub-block.
 6. The method of claim5, wherein the specific position is a center of the correspondingsub-block.
 7. The method of claim 6, wherein the corresponding sub-blockhas a size M×N and the center is defined as [(M>>1)+a, (N>>1)+b],wherein M and N are natural numbers and a, b is 0 or −1.
 8. The methodof claim 1, further comprising: determining a CPMV candidate of thecurrent video block to be derived from motion vectors from top adjacentblocks coded with the affine mode based on a position of the currentvideo block.
 9. The method of claim 1, wherein the conversion includesencoding the current video block into the bitstream representation. 10.The method of claim 1, wherein the conversion includes decoding thecurrent video block from the bitstream representation.
 11. An apparatusfor processing video data comprising a processor and a non-transitorymemory with instructions thereon, wherein the instructions uponexecution by the processor, cause the processor to: determining, for aconversion between a current video block of a video and a bitstreamrepresentation of the video, motion vectors at control points (CPMV) ofthe current video block based on a rule, wherein the rule specifies toexclude using of non-adjacent neighboring block from one or moreneighboring blocks of the current video block; and performing theconversion between the current video block and the bitstreamrepresentation based on the motion vectors.
 12. The apparatus of claim11, wherein the rule further specifies to exclude using of invalidneighboring block from the one or more neighboring blocks based onpositions of the one or more neighboring blocks.
 13. The apparatus ofclaim 12, wherein the current video block belongs to a current slice,and a neighboring block of the one or more neighboring blocks is invalidin a case that the neighboring block belongs to a slice different fromthe current slice.
 14. The apparatus of claim 12, wherein the currentvideo block belongs to a current tile, and a neighboring block of theone or more neighboring blocks is invalid in a case that the neighboringblock belongs to a tile different from the current tile.
 15. Theapparatus of claim 11, wherein the current video block includes multiplesub-blocks, and performing the conversion comprising: determining amotion vector for each sub-block of the multiple sub-blocks based on theCPMV and a specific position of the corresponding sub-block.
 16. Theapparatus of claim 15, wherein the specific position is a center of thecorresponding sub-block.
 17. The apparatus of claim 16, wherein thecorresponding sub-block has a size M×N and the center is defined as[(M>>1)+a, (N>>1)+b], wherein M and N are natural numbers and a, b is 0or −1.
 18. The apparatus of claim 11, further comprising: determining aCPMV candidate of the current video block to be derived from motionvectors from top adjacent blocks coded with the affine mode based on aposition of the current video block.
 19. A non-transitorycomputer-readable storage medium storing instructions that cause aprocessor to: determining, for a conversion between a current videoblock of a video and a bitstream representation of the video, motionvectors at control points (CPMV) of the current video block based on arule, wherein the rule specifies to exclude using of non-adjacentneighboring block from one or more neighboring blocks of the currentvideo block; and performing the conversion between the current videoblock and the bitstream representation based on the motion vectors. 20.A non-transitory computer-readable recording medium storing a bitstreamrepresentation which is generated by a method performed by a videoprocessing apparatus, wherein the method comprises: determining, for aconversion between a current video block of a video and a bitstreamrepresentation of the video, motion vectors at control points (CPMV) ofthe current video block based on a rule, wherein the rule specifies toexclude using of non-adjacent neighboring block from one or moreneighboring blocks of the current video block; and generating thebitstream representation from the current video block based on thedetermining.